EP 1 720 131 B1 describes an augmented reality system with real marker object identification. The system comprises a video camera for gathering image data from a real environment. The real environment represents any appropriate area, such as a room of a house, a portion of a specific landscape, or any other scene of interest. The real environment represents a living room comprising a plurality of real objects for instance in the form of walls and furniture. Moreover, the real environment comprise further real objects that are considered as marker objects which have any appropriate configuration so as to be readily identified by automated image processing algorithms. The marker objects have formed thereon significant patterns that may easily be identified, wherein the shape of the marker objects may be designed so as to allow identification thereof from a plurality of different viewing angles. The marker objects also represent substantially two-dimensional configurations having formed thereon respective identification patterns.
The system further comprises a means for identifying the marker objects on the basis of image data provided by the camera. The identifying means may comprise well-known pattern recognition algorithms for comparing image data with predefined templates representing the marker objects. The identifying means may have implemented therein an algorithm for converting an image, obtained by the camera, into a black and white image on the basis of predefined illumination threshold values. The algorithms are further configured to divide the image into predefined segments, such as squares, and to search for pre-trained pattern templates in each of the segments, wherein the templates represent significant portions of the marker objects.
First the live video image is turned into a black and white image based on a lighting threshold value. This image is then searched for square regions. The software finds all the squares in the binary image, many of which are not the tracking markers, such as the objects. For each square, the pattern inside the square is matched against some pre-trained pattern templates. If there is a match, then the software has found one of the tracking markers, such as the objects. The software then uses the known square size and pattern orientation to calculate the position of the real video camera relative to the physical marker such as the objects. Then, a 3×4 matrix is filled with the video camera's real world coordinates relative to the identified marker. This matrix is then used to set the position of the virtual camera coordinates. Since the virtual and real camera coordinates are the same, the computer graphics that are drawn precisely superimpose the real marker object at the specified position. Thereafter, a rendering engine is used for setting the virtual camera coordinates and drawing the virtual images.
The system further comprises means for combining the image data received from the camera with object data obtained from an object data generator. The combining means comprise a tracking system, a distance measurement system, and a rendering system. Generally, the combining means is configured to incorporate image data obtained from the generator for a correspondingly identified marker object so as to create virtual image data representing a three-dimensional image of the environment with additional virtual objects corresponding to the marker objects. Hereby, the combining means is configured to determine the respective positions of the marker objects within the real environment and also to track a relative motion between the marker objects with respect to any static objects in the environment and with respect to a point of view defined by the camera.
The system further comprises output means configured to provide the virtual image data, including the virtual objects generated by the generator wherein, in preferred embodiments, the output means is also configured to provide, in addition to image data, other types of data, such as audio data, olfactory data, tactile data, and the like. In operation, the camera creates image data of the environment, wherein the image data corresponds to a dynamic state of the environment which is represented by merely moving the camera with respect to the environment, or by providing moveable objects within the environment; for instance the marker objects or one or more of the objects are moveable. The point of view of the environment is changed by moving around the camera within the environment, thereby allowing to observe especially the marker objects from different perspectives so as to enable the assessment of virtual objects created by the generator from different points of view.
The image data provided by the camera, which are continuously updated, are received by the identifying means, which recognizes the marker objects and enables the tracking of the marker objects once they are identified, even if pattern recognition is hampered by continuously changing the point of view by, for instance, moving the camera or the marker objects. After identifying a predefined pattern associated with the marker objects within the image data, the identifying means inform the combining means about the presence of a marker object within a specified image data area and based on this information, the means then continuously track the corresponding object represented by the image data used for identifying the marker objects assuming that the marker objects will not vanish over time. The process of identifying the marker objects is performed substantially continuously or is repeated on a regular basis so as to confirm the presence of the marker objects and also to verify or enhance the tracking accuracy of the combining means. Based on the image data of the environment and the information provided by the identifying means, the combining means creates three-dimensional image data and superimposes corresponding three-dimensional image data received from the object generator, wherein the three-dimensional object data are permanently updated on the basis of the tracking operation of the means.
The means may, based on the information of the identifying means, calculate the position of the camera with respect to the marker objects and use this coordinate information for determining the coordinates of a virtual camera, thereby allowing a precise “overlay” of the object data delivered by the generator with the image data of the marker objects. The coordinate information also includes data on the relative orientation of the marker objects with respect to the camera, thereby enabling the combining means to correctly adapt the orientation of the virtual object. Finally, the combined three-dimensional virtual image data is presented by the output means in any appropriate form. The output means may comprise appropriate display means so as to visualize the environment including virtual objects associated with the marker objects. When the system is operated, it is advantageous to pre-install recognition criteria for at least one marker object so as to allow a substantially reliable real-time image processing. Moreover, the correlation between a respective marker object and one or more virtual objects may be established prior to the operation of the system or is designed so as to allow an interactive definition of an assignment of virtual objects to marker objects. For example, upon user request, virtual objects initially assigned to the marker object are assigned to the marker object and vice versa. Moreover, a plurality of virtual objects is assigned to a single marker object and a respective one of the plurality of virtual objects is selected by the user, by a software application.
The object of the invention is to improve a system for a motor vehicle.
Said object is attained by a system with the features of independent claim 1. Advantageous refinements are the subject of dependent claims and are included in the description.
Accordingly, a system for a vehicle is provided. The system can also be called an infotainment system, if it has both information and entertainment functions.
The system has a head-up display.
The system has a circuit which is connected to the head-up display. The circuit may be integrated into a central unit. Alternatively the circuit may be integrated into a housing having the head-up display.
The system has recording means for recording first image data of the vehicle surroundings. The recording means is connected to the circuit.
The circuit is configured to recognize an object in the surroundings based to the recorded first image data. This functionality may also be referred to as object recognition.
The head-up display is configured to project an image onto a windshield of the vehicle or onto a combiner in the driver's field of view. The combiner may be distanced from the windshield. The combiner is a reflecting at least partly transparent pane, e.g. made of transparent plastic material.
The circuit is configured to generate second image data for outputting the image and to send them to the head-up display.
The circuit is configured to generate a virtual barrier in the second image data.
The circuit is configured to position the virtual barrier within the image based on a determined position of the recognized object, whereby the positioned virtual barrier in the driver's view direction to the recognized object overlaps the recognized object at least partially.
Tests by the applicant have shown that intuitively understandable warning information can be output to the driver by the virtual barrier. Thus, the driver need not first interpret a symbol and associate the interpreted symbol with a source of danger. The interpretation and association with danger are taken over by the virtual barrier positioned in the field of view.
Advantageous embodiments of the system will be described below.
According to one embodiment, the virtual barrier may be a virtual wall. The driver would like to avoid collision with a wall instinctively, so that the driver can countersteer rapidly and nearly reflexively and gain valuable fractions of a second of time before a possible collision.
According to one embodiment, the virtual barrier may be semitransparent. Because of the transparency of the virtual barrier the driver can recognize real objects through the virtual barrier, for example, a moving trunk.
According to one embodiment, the recognized object may be a road marking. The road marking may be a side line or a center line in white or yellow.
According to one embodiment, the recognized object may be a traffic sign. The traffic sign may be a “no entry” placed at the exit ends of a one-way street. The virtual barrier may overlap the traffic sign and virtually closing the exit end of the one-way street, so that the driver is hindered to drive in the wrong direction virtually.
According to one embodiment, the recognized object may be a constructional boundary of a road. The constructional boundary may be a kerb stone edge. The virtual barrier may overlap the kerb stone edge.
According to one embodiment, the circuit may be configured to generate the virtual barrier when a warning signal exceeds a threshold. According to one embodiment, the circuit may be configured to generate the warning signal based on a determined distance of the vehicle to the object. Alternatively, the warning signal can also be generated depending on other input variables, for example, traffic rules or the like.
According to one embodiment, the circuit may be configured to recognize a road user. The circuit may be configured to generate the warning signal depending on the recognition of the road user in the blind spot.
The previously described embodiments are especially advantageous both individually and in combination. In this regard, all embodiments can be combined with one another. Some possible combinations are explained in the description of the exemplary embodiments shown in the figures. These possible combinations of the features depicted therein, are not definitive, however.